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ABSTRACT . 

A study was sponsored by the U.S. National Educational 
Research Policy and Priorities Board of the Office of Educational Research 
and Improvement (OERI) amid the enthusiasm for bringing the power of 
rigorous, objective, scientific understanding to bear on improving decisions 
about educational programming and thus student achievement. There is 
controversy and lack of consensus about what it actually means for something 
to be based on "scientific research" in education. Although this paper does 
not talk about the findings related to the research agency, it adopts a 
forward-looking approach: describing the enterprise in the ideal and 
highlighting its successes. The paper offers no hard-and-fast definition of 
what constitutes scientific research in education and states that there will 
rarely be any one study that should be' taken as the definitive "answer" to 
questions about education. It outlines and explains the following six 
principles of scientific inquiry: (1) Pose significant questions that can be 

investigated empirically; (2) Link research to theory; (3) Use methods that 
permit direct investigation of questions; (4) Provide coherent chain of 
rigorous reasoning; (5) Replicate and generalize; and (6) Transparency and 
scholarly debate. The paper notes that education research is most closely 
associated with the social and behavioral sciences, and it provides some 
information on the special characteristics of education research, (NKA) 
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Submitted Paper — ^The Logic and the Basic Principles of 
Scientific Based Research — Michael Feuer and Lisa Towne 
Scientifically Based Research 



Our presentation today is based on a recently Related documents 
released study authored by the National _ jranscript 

Research Council's Committee on Scientific “ 

Principles in Education Research. That report can be read online for free 
and additional hard copies are available for sale at 
htt p ://www. n a D . edu/cata loa /10236.html . 

Context 

The study was sponsored by the U.S. National Educational Research 
Policy and Priorities Board of OERI amid the very interesting context that 
brings us all here today: the simultaneous enthusiasm for bringing the 
power of rigorous, objective, scientific understanding to bear on improving 
decisions about educational programming and thus student achievement, 
and the controversy and lack of consensus about what it actually means for 
something to be based on "scientific research" in education. As most of you 
know, on the one hand, before No Child Left Behind there was the 
Comprehensive School Reform Demonstration Act and the Reading 
Excellence Act — two federal education programs driven by the desire to 
ground program decisions in the best available evidence of their 
effectiveness. NCLB extends that reach with its myriad references to 
"scientifically based research" across the full range of programs the act 
covers. 

At the same time, you are probably aware that not only does there not 
seem to be any working consensus on what exactly is meant by scientific 
research in education, but that there is also deep skepticism about the 
quality of existing work available for decision makers in complying with this 
requirement. Some of you may be aware that in the summer of 2000, 
Representative Castle (R-DE) introduced a bill to reauthorize OERI that 
included definitions of "scientifically valid quantitative methods" and 
"scientifically valid qualitative methods". While elected officials have long 
engaged in the federal research effort in establishing priorities, rarely if ever 
have they instructed researchers on the tools of their trade and codified it in 
federal statute. This report was intended to engage a group of prominent 
researchers to articulate the nature of their work. 

It is important to make a few introductory notes about what the committee- 
and thus the report-did and did ndt do. Without detailing the exact charge 
or how the committee went abol^t fulfilling it, we do want to underscore a 
few points relevant to today's' discussion. First, the committee was asked to 
describe the principles of scientific research in education as well as to draw 
out the implications of those findings for the future of a federal education 
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research agency. We will not be talking about the findings related to the 
research agency today, but are hopeful that they can inform the pending 
reauthorization of OERI. Also, It is important to understand that the 
committee did not evaluate the quality of existing research in the field. We 
will no doubt be hearing about the quality of existing research with respect 
to particular program areas later this morning. This report, and thus our 
remarks this morning, adopt a forward-looking approach: describing the 
enterprise in the ideal and highlighting its successes. 

Key Messages 

If you were hoping to come to this meeting to get a hard-and-fast definition 
of what constitutes scientific research in education, your expectations will 
not be fulfilled. There is no algorithm for science, nor is there a checklist for 
how to evaluate its quality. Of course. No Child Left Behind had to include 
definitions, but no statute could adequately define science in education or 
otherwise. Philosophers of science and researchers have been debating 
such questions for hundreds of years. The main point here is not that we 
haven't figured out the algorithm yet, but rather that it does not exist; 
science is in part a creative enterprise. 

The other key point is that there will rarely be any one study that should be 
taken as the definitive "answer" to questions about education. Science, by 
definition, is an uncertain enterprise that evolves over time. Things we take 
for granted as true (e.g., the earth is flat) can be completely reversed by 
subsequent inquiry. And science progresses as individual studies and their 
findings are integrated into current understanding. The NRC committee 
report emphasizes this accumulation of knowledge, arguing that it is 
sustained inquiry over time that produces insights, not typically any single 
investigation. 

Although it is creative, science is also a very disciplined enterprise that is 
supported by norms and practices, what the NRC committee called a 
"culture of Inquiry." It is these norms, or principles, that we will describe 
today. 

Guiding Principles of Scientific Inquiry 

These norms apply to all sciences — to cell biology, ecology, economics, 
developmental psychology, and scientific education research. Although all 
sciences share these principles, the way they are applied varies depending 
on the objects of study (e.g., schools) and the context in which they are 
studied (e.g., highly mobile populations of students, multiple languages 
spoken). Thus, we will also briefly describe some of the features of 
education that shape the ways in the principles play out in the scientific 
study of education. 

Principle 1: Pose Significant Questions That Can Be Investigated 
Empirically 

The committee emphasizes here the idea that simply asking a question in a 
new way can lead to scientific breakthrough. The significance of the 
question relates to, for example, the extent to which inquiries inform core 
problems in education, or builds on prior knowledge. This does not mean 
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that basic research is irrelevant: fundamental research in neuroscience, for 
example, has clear implications for how we educate our children. This 
underscores an important point that program evaluation and more 
specifically the estimation of the effects of a particular intervention are only 
one part of a larger research base that is potentially useful for informing 
policy and practice. What makes a question significant, then, can derive 
both from the practical problems of teaching, learning, and schooling as 
well as the state of knowledge in a particular area. 

The word empirical basically means observation, and the use of this word 
simply signals that science can only address questions that can be 
answered through systematic investigation or observation. Some important 
questions lie solely in realms outside of science (e.g. should students be 
required to recite the pledge of allegiance each school day?). 

Principle 2: Link Research to Theory 

Much of science is fundamentally concerned with developing and testing 
theories that can help explain some aspect of the world. Evolution and 
quantum theory are examples of well-known theories. In the social 
sciences and education, such "grand" theories are rare, but the goal is still 
theoretical understanding. An important point here is that data are used in 
the process of scientific inquiry to relate to a broader framework that drives 
the investigation. Data about achievement or school spending alone are not 
useful in a scientific investigation unless they are explicitly used to address 
a specific question with a specified theoretical model or to generate a 
theory or conjecture that can be tested later. Even in program evaluations, 
program developers have at least an implicit conceptual model in mind of 
how a particular program is supposed to achieve its objectives, and thus 
this theoretical frame drives the evaluation. 

Principle 3: Use Methods That Permit Direct Investigation of Question 

Methodology is a key feature of science, but it does not uniquely define it. 
The method or design used in a particular investigation does not itself 
make the study scientific, and methods in the abstract cannot be judged to 
more or less scientific either. There is a wide range of legitimate methods 
available to researchers in all fields-the NRC report demonstrates this 
diversity of method in several examples inside and outside education 
research. 

More specifically, it is often the case that the use of multiple methods can 
significantly strengthen the certainty with which conclusions can be drawn. 
Think of this idea as being an extension of the more general notion of 
thinking about a problem from a number of different perspectives: if you 
can convince yourself that a particular course of action is ideal from several 
angles, your confidence that it is the "right" thing to do increases. 

The last point related to method is that some methods are better than 
others for particular purposes. Thus, the quality of a particular method can 
only be judged with respect to how well it addresses the question at hand. 
The NRC report provides a good bid of detail and several examples of this 
idea. The committee described a set of common questions in education 
research and discusses the most commonly used methods for addressing 
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them under various conditions. So for example, when the research base on 
a particular area is weak-like in the case of understanding how children 
come to learn the mathematical concepts of ratio and proportion-in-depth, 
longitudinal, qualitative methodologies will likely be the most appropriate to 
start to develop theoretical models of student learning of these ideas. On 
the other hand, when investigators are trying to estimate the effectiveness 
of a fairly well defined educational intervention-say, for example, a 
comprehensive school reform model-the use of random assignment is 
especially well suited. To reiterate the earlier point about the use of multiple 
methods, it is also true that such evaluations of programs using random 
assignment methodologies are often significantly strengthened by 
qualitative methods that focus on what is happening inside classrooms. 
These tools can often help researchers identify and rule out alternative 
6xp!3n3tior!s for why studont 3chl6V6rn6nt m3y bo difforont in c!3ssrooms 
receiving the intervention versus others. 



Principle 4: Provide Coherent Chain of Rigorous Reasoning 

This is largely what Valerie talked about in the previous presentation: the 
logic behind scientific reasoning. Again, there is no one, linear way to 
reason scientifically, but in general terms it must be coherent, explicit, and 
persuasive to the skeptical reader. The process of reasoning is conducted 
to produce what John Dewey called a "warrant" — a scientific justification — 
for inferences and conclusions. And it is important to point out that this logic 
is fundamentally the same for both quantitative and qualitative research. 
Scientific reasoning is characterized by clearly stating the assumptions 
present in the analysis, how evidence was judged to be relevant, how data 
relate to theoretical conceptions, how much error or uncertainty is 
associated with conclusions, and perhaps most importantly, how alternative 
explanations for what was observed were treated. 



Principle 5: Replicate and Generalize 

Scientific inquiry emphasizes checking and validating individual findings 
and results in different times, places, and contexts. Since all studies rely on 
a limited number of observations, a key question is how scientific 
inferences-that is, the conclusions of scientific work-generalize to a broader 
population or setting. Successfully replicating findings in different contexts 
can strengthen a theory or working consensus over time. In education 
research, contextual factors often are very important. Teachers and 
researchers alike have long noted that a particular program that works in 
one classroom may not replicate in a classroom just down the hall or within 
the same classroom but with a different group of students the next year. In 
education research, then, attention to the conditions under which a 
particular classroom being studied is quite important in understanding the 
extent to which findings will generalize beyond it. 



Principle 6: Transparency and Scholarly Debate 

A final principle of science relates directly to the "culture of inquiry" we 
described earlier. Researchers must engage in ongoing scrutiny of each 
other's work: by publishing in peer reviewed journals, presenting findings at 
conferences, and the like. Educators often bemoan what they perceive as 
bickering within the research community as evidence that the community 
has somehow failed. On the contrary, researchers are trained and 
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employed to ask critical questions and be skeptical observers. Of course, 
they also engage in such critique to try and forge consensus about the 
current state of knowledge in a particular area. The community of 
researchers has to collectively make sense of new findings to integrate 
them into the existing corpus of work. Indeed, the objectivity of science 
derives from these self-enforced norms, not the attributes of a particular 
person or method. 

How is Education Research Speciai? 

We have argued that the preceding principles apply to all sciences. Here, 
we provide a flavor for how these principles get applied in education 
research. Education research is most closely associated with the social and 
behavioral sciences, so some broad differences between the physical and 
social sciences help to understand the nature of education research. 

For example, researcher control is often stronger in the physical sciences. 
Think of it this way: a Petri dish of heart cells is typically better behaved 
than a classroom of third-graders! The role of the researcher in sometimes 
different in hard vs. soft sciences-in the natural sciences it is customary for 
the researcher to be removed from the process of inquiry so as to minimize 
bias; in the social sciences, in some cases is not desirable. Also, theory in 
the social sciences tends to be used to model past behavior rather than to 
predict the future. And finally, it is very important to recognize that the level 
of uncertainty is typically higher when studying humans as compared to 
studying inanimate objects. All sciences have a degree of uncertainty 
associated with them, but our theoretical understanding of human behavior 
is still pretty elementary. That is why estimating the certainty of results is so 
important in education research. And this is important for consumers of 
research to understand as well: research can be an incredibly powerful tool 
for helping to make practical decisions, but rarely will it ever "prove" beyond 
a doubt that one strategy or another will be successful. 

In education specifically, the NRC committee briefly described five features 
of education that influence research in it. For example, it talked about the 
proper role of values in making education decisions in our democracy and 
its influence on things like the choice of what is studied and how findings 
are interpreted and used. 

Human volition is another factor. People are complex beings who often 
have priorities that may not comport with those of researchers trying to 
study them. This can result in samples changing over time due to high 
rates of student mobility or parents taking their children out of a particular 
program to which they were assigned for research purposes. 

The local control of schools also means that the nature of programs — even 
those that are called the same thing — can be implemented very differently 
across the country and can change substantially year to year. Anyone who 
has ever tried to evaluate federal programs knows the issue of "fidelity" is 
critical to understanding its impact. Here again, the point is that paying 
close attention to this context is important in doing research. The 
hierarchical nature of schools means that for researchers to understand 
what is going on in a particular school, they typically must study it with a 
good understanding of what is going on at the district, state, and even 
federal levels that influence it. Finally, the cultural, language, racial, ethnic. 
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and geographic diversity that characterizes our nation also, of course, 
characterizes people in education institutions. And programs may work for 
some populations but not others, so researcher must explicitly attend to 
these factors. 

Finally, education research itself is characterized by multiple disciplines — 
for example: developmental psychologists study fundamental processes of 
cognition, language and socialization; economists study the incentive 
structures of schools and their relationship to behavior; political scientists 
study the implementation of large-scale institutional changes, like charter 
schools. That means again that lots of different methods are used in 
education research and that the challenge is to integrate what is known 
from each of these perspectives into some shared understanding. 

Also, education research is sometimes curtailed due to justifiable ethical 
considerations to ensure proper treatment of children (although education 
research rarely presents any risk to research participants). Finally, 
education research depends critically on its relationships to educational 
practitioners. Researchers typically at least need the cooperation of 
schools and students to conduct their work and increasingly practitioners 
are entering into full partnership with researchers. 
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